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OVERVIEW 


oo many talented students who go to under-resourced schools do not achieve their full potential. 

Though they may perform very well relative to their classmates, these students do not receive the 
same kinds of academically challenging opportunities throughout their educational journey as do their 
counterparts in better-resourced public and private schools. Rather than matriculating to competitive 
high schools and from there to selective colleges they are qualified to attend, these students often go 
to less academically competitive high schools and on to colleges where graduation rates are low. Some 
even forgo college altogether. 


To address this problem, Higher Achievement offers an intensive, academically oriented program for 
middle school students in under-resourced schools. Starting in the summer before fifth or sixth grade, 
Higher Achievement offers its participants, called “scholars,” 650 extra hours of academic enrichment 
and instruction after school and during the summers through the eighth grade. The program includes 
English and math instruction as well as field trips to competitive high schools and colleges, achievement 
test preparation, and assistance in applying for financial aid. This short report presents the results of 
a randomized controlled trial of Higher Achievement that started in 2005, comparing the outcomes of 
students who were offered the opportunity to participate in Higher Achievement (the program group) 
and students who were not (the control group). It presents the impacts of the program one, two, and 
four years after enrollment, as well as its long-term impacts on college attendance. 


The study found that Higher Achievement was successful at changing the educational trajectory of 
students through middle school and improved the academic quality of many students’ high school 
experiences, but did not affect the colleges to which they matriculated. By Year 2, there were positive 
impacts on students’ math and reading test scores. In Year 4, the impacts on math test scores remained 
statistically significant. Higher Achievement had a small impact on the types of high schools its scholars 
ultimately attended. Program group students were more likely than control group students to 
matriculate to private or parochial schools and less likely to go to nonacademically competitive 
charter or magnet schools. By 2019, there was no difference in college going. More than 70 percent 
of both program and control group students had ever attended college. There were no differences in 
the academic quality of the colleges Higher Achievement’s scholars ultimately attended, as measured 
by being a two- or four-year college; a college having a lower acceptance rate; or a college whose 
freshmen on average had higher SAT math or reading scores. Higher Achievement’s college impacts 
did not differ by whether a student’s parent had attended college or by student characteristics. 


The study shows that Higher Achievement is a very effective middle school program, improving stu- 


dents’ middle school trajectories. However, the impacts did not persist after the program through high 
school and college. 
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INTRODUCTION 


Too many talented students who go to under-resourced schools do not achieve their full potential. 
Though they may perform very well relative to their classmates, these students do not receive 
the same types of academically challenging opportunities throughout their educational journey 
as do their counterparts in private and better-resourced public schools. The high schools they 
subsequently attend also have fewer trained college counseling staff, so these students and 
their families generally receive less assistance with the college application process. As a result, 
academically motivated students from disadvantaged backgrounds frequently “undermatch” 
when they apply to college: Rather than matriculating to competitive, selective colleges that they 
are qualified to attend, they frequently attend much less competitive or nonselective schools 
where graduation rates are low. Some even forgo college altogether." 


It is unlikely that any one program can address all of the inequities underlying this problem. 
But starting in 1975, Higher Achievement began targeting the particularly crucial time between 
the end of elementary school and the end of middle school, when many students leave the path 
that would ultimately have led them to college.? Higher Achievement’s goal is to enable students 
to attend college-preparatory public and private high schools — schools that will help them get 
into selective, academically stronger colleges. 


Starting in the summer before fifth or sixth grade, Higher Achievement offers its participants, 
called “scholars,” hundreds of hours of academic enrichment and instruction after school and 
during the summers through the eighth grade. Scholars attend the program’s Achievement 
Centers, located in select low-income middle schools. Unlike many academic after-school and 
summer programs in under-resourced schools, Higher Achievement is not a remedial program 
targeting struggling students. Rather, the program challenges motivated scholars to meet the 
high academic standards expected of college-bound students. It uses a diverse community of 
adults — trained volunteers called “mentors,” part-time staff called “achievement coaches,” center 
aides, and summer teachers — to deliver homework help, enrichment activities, and academic 
instruction after school and during the summer. 


An evaluation of DC Metro, the original Higher Achievement program in Washington, DC, and 
Alexandria, Virginia, began in 2005.° This report describes the program as it operated then; 
summarizes its impacts one, two, and four years after enrollment; and presents its long-term 
impacts on college attendance. 


1 Bowen, Chingos, and McPherson (2009); Byndloss and Reid (2013); Hoxby and Turner (2013); Roderick, 
Nagaoka, and Coca (2009). 


Gutman and Midgley (2000); Seidman et al. (1994). 


Herrera, Linden, Arbreton, and Grossman (2011b). 


HIGHER ACHIEVEMENT DURING THE STUDY: 
BACKGROUND AND SERVICES 


A decade and a half ago, most after-school programs offered nonacademic enrichment three 
hours a day, five days a week, and middle school students typically participated one or two days 
a week.* Academic instruction was often remedial in nature. Rigorous evaluations of these after- 
school programs rarely found positive academic impacts, and summer school programs, if they 
worked at all, served and benefited only those students who were behind in school, rather than 
those trying to excel.° Given this landscape, Higher Achievement, founded in 1975, stood out as 
an unconventionally intensive and academically rigorous program in the communities it served. 


While Higher Achievement continues to expand and reshape its programming, this report pres- 
ents the program as it was when the study began in 2005. Then and now, Higher Achievement 
targeted academically motivated students and their parents, who completed an application and 
met with Higher Achievement staff. Family members were interviewed separately to ensure 
that the student was excited about participating in such a time-intensive, academically rigorous 
program and that the family member understood the multiyear commitment of getting their 
child to and from the program’s activities. 


At the time of the study, Higher Achievement offered its scholars up to 650 hours annually of 
academic enrichment through its Summer and Afterschool Academies. The six-week Summer 
Academy operated from 8 a.m. to 4 p.m., five days a week. Scholars took math, science, social 
studies, and literature classes in the morning, and two recreational or enrichment electives such 
as soccer or drama after lunch. The morning classes, typically about 13 students per grade level 
and taught by paid teachers, focused on academic concepts the children would be exposed to 
during the coming school year. Scholars also took weekly field trips, participated in academic 
competitions, and participated in a three-day university trip, during which they got the chance 
to experience college life — attending classes, sleeping in dorms, going to lectures, and eating 
in dining halls. 


During the school year, the Afterschool Academy was offered three days a week from 3:30 p.m. 
to 8 p.m. for 25 weeks. Each four-and-a-half-hour session included homework help, dinner, an 
arts or recreation elective, a 25-minute community gathering (attended by all staff, volunteer 
mentors, and participating students), and 75 minutes of small-group academic instruction led by 
mentors using a structured curriculum. These mentors worked with their groups for one day a 
week and committed to stay with the program through at least one school year. Each scholar thus 
had three mentors — one who delivered the math curriculum, one for the literature curriculum, 
and one who led a seminar that included such topics as creative writing, conflict resolution, and 
technology. The curricula that guided these mentoring groups followed skill standards set by 
the Washington, DC, and Alexandria public school systems. Scholars also had the opportunity 
to take part in monthly field trips, career-shadowing days, and community service projects. 


4 Dynarski et al. (2003); Grossman et al. (2002). 
5 Dynarski et al. (2003). 
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As students approached high school, Higher Achievement increased its focus on preparing them 
to apply to college-preparatory high schools. At the end of the seventh grade, staff members held 
a family night that included a group discussion with parents about high school applications. 
During the Summer Academy, rising eighth-graders spent two days visiting quality high schools 
and engaging in guided discussions about these schools. To place scholars into competitive, 
academically engaging high schools, the Afterschool Academy offered eighth-graders a third 
program component, the High School Placement Program, delivered one day a week. It com- 
prised a 10-week test preparation class; application and financial aid workshops; school visits; 
and individualized support for families as they navigated the high school admissions process.°® 
Higher Achievement staff also helped students select schools and hone their interviewing skills. 
Staff met individually with each eighth-grader’s parent to review the student’s grades, answer 
questions, and make recommendations for high school. The manager of school placement from 
the DC Metro office also made biweekly visits to each center to answer individual scholars’ 
questions and help with the application process. In other words, participants and their families 
learned how to choose a set of schools to apply to, complete the application process, and apply 
for financial aid — skills that would assist them not only with applying to high school but also 
later on with the college application process. The High School Placement Program is the program 
component that is most closely linked to potential college-going effects. 


DESIGN OF THE EVALUATION 


The impacts of Higher Achievement’s DC Metro program were evaluated using a random 
assignment research design. In the spring of 2006, 2007, and 2008, rising fifth- and sixth-graders 
whose families still wanted to participate after meeting individually with Higher Achievement staff 
were invited to take part in a randomized controlled trial to assess the program’s impacts. If the 
students and their parents agreed to participate, students completed standardized tests in reading 
and math and completed a survey;’ their parents were also surveyed. Then a lottery was used to 
determine which students would be offered the opportunity to participate in Higher Achievement 
(the program group) and which would not (the control group). This random assignment design 
ensured that at the start of the study the two groups were, on average, identical, including the 
students’ and their families’ academic motivation, and that any differences that emerged between 


6 If students wished to attend a high school other than their neighborhood school — be it a private school, a 
public charter school, a public magnet school, or a parochial school — they had to apply individually to the 
ones they wanted. Many schools require students to take a standardized high school entrance exam and 
be interviewed. Charter and magnet schools are public and thus are free for students who are admitted. 
Private and parochial schools charge tuition. To offset some or all of the tuition, parents have to submit 
requests for financial aid as well as documentation of the family’s income, to the relevant schools. 


7 To assess both reading and math skills in less than three hours (to limit the burden on families), the 
evaluators could not administer the full battery of reading (vocabulary and reading comprehension) and 
math (math procedures and problem-solving) questions. Instead, they had sample members take only 
the reading comprehension and math problem-solving Stanford Achievement Test 10 subtests. These 
subtests, respectively, required vocabulary knowledge and math procedures knowledge and thus were 
relatively comprehensive. 
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the groups over time (the impacts) could be attributed to the Higher Achievement program.® 
Table 1 shows the characteristics of the two groups when they applied. Table 2 shows how the 
program students progressed through Higher Achievement over time. 


After the initial assessment (the baseline), the research team retested all study students in 
reading comprehension and math problem-solving, and resurveyed both groups of youth and 
their parents several times: 


e Inthe spring — one, two, and four years after random assignment (Years 1, 2, and 4) 


e In the fall of 2010 — when the team conducted a survey focused primarily on the 
students’ learning and experiences during the previous summer 


At each point, the surveys measured social-emotional attitudes related to learning, such as the 
students’ perceptions of their own academic abilities, persistence, creativity, and curiosity, as 
well as their participation in a range of enrichment activities. In the Year 4 survey, the research 
team also asked parents and youth about the steps they had taken in the high school application 
process, such as preparing for and taking entrance exams, visiting high schools, doing mock 
interviews, and completing financial aid forms. Parents were asked what schools their child had 
applied to, been accepted into, and ultimately planned to or did attend. 


EARLY FINDINGS 


Three reports have been written about this study so far.? These reports show that Higher 
Achievement is a successful middle school program. First, unlike many programs targeting 
middle school students, Higher Achievement retains a large percentage of its enrollees. A survey 
of 198 after-school programs in six cities found that, on average, they retained only 22 percent 
of their middle-school-age participants for a year or more.'° Higher Achievement retained 70 


8 The study used an “intent to treat” impact design — the most rigorous and conservative approach 
available to researchers. At each wave of data collection, the research team contacted all of the youth 
in the sample that could be contacted, both control and program students, including all program youth 
regardless of whether they had actually attended the program. Using regression analysis, the outcomes of 
the originally assigned program group were compared with those of the originally assigned control group. 
The difference was deemed to be a statistically significant impact if that difference was large enough to 
not have occurred by chance with a probability of 0.1 or less (p<0.1). Technically, the study addressed 
the question of whether having access to the program affected youth outcomes, not whether program 
participation, per se, affected outcomes. In this way, the impacts reported here likely understate Higher 
Achievement’s effects on those youth who fully participated in the program. 

9 Herrera, Linden, Arbreton, and Grossman (2011b) summarizes findings from Years 1 and 2; Herrera 
Linden, Arbreton, and Grossman (2011a) describes the program’s effect on learning during the summer of 
2011; Herrera, Grossman, and Linden (2013) summarizes the program’s academic and high-school-choice 
impacts four years after enrollment. 


10 Deschenes et al. (2010). 
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TABLE 1 Baseline Comparison of Student Characteristics 
by Research Group 


CONTROL ESTIMATED PROGRAM- 


STUDENT CHARACTERISTICS AVERAGE CONTROL DIFFERENCE 
Demographic characteristics 

Female 0.59 < 0.01 
Grade 5.42 < 0.01 
Age 9.83 0.05 
African-American 0.76 -0.02 
Latino 0.76 < 0.01 
Free or reduced-price Lunch 0.66 -0.05 
Normalized test scores 

Reading 0.10 < 0.01 
Problem-solving 0.10 < 0.01 


Household characteristics 
Total household income 


Below $25,000 0.32 -0.03 

$26,000-$50,000 0.38 -0.03 

$51,000-$75,000 0.16 -0.03 

Over $75,000 0.14 -0.04 
Parent education 

College degree 0.33 -0.01 

Some college 0.31 < 0.01 
Living with both parents 0.41 -0.04 
Non-English language spoken at home 0.17 < 0.01 
Number of observations 951 


SOURCE: Calculations based on information collected from either the Higher 
Achievement application or the baseline survey. 


NOTE: This table provides a comparison of the baseline characteristics of the 

521 students assigned to the program group and the 430 assigned to the control 
group. Column one contains the average characteristics of the control group. Column 
two provides the estimated difference between the program and control group after 
controlling the baseline variables used in the impact regressions. These covariates 
include child-level variables: baseline scores for reading comprehension and 
problem-solving; self-perceptions of academic abilities; industry and persistence; 
creativity, enjoyment of learning, curiosity, and ability to change the future through 
effort; peer academic support and general adult support; indicator variables for grade 
at baseline, age, receipt of free or reduced-price lunch, race, household language; 
and a dummy variable for the year the child applied to Higher Achievement and 
whether the student applied with a sibling. In addition, there are family-level controls 
including fixed effects for the center to which the family applied, parent’s education, 
household income, household composition, and an indicator for whether English 

is the primary language spoken at home. None of these estimated differences is 
statistically significant at a 0.10 level. 
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percent of its scholars for two years after they were admitted to the program." However, four 
years after being admitted, only 47 percent of their scholars were still attending.’? Thus, only 
about half of the scholars received the full High School Placement component of the program 
that was delivered only in the eighth grade — the last year of the program. 


Second, Higher Achievement had sustained impacts on students’ math outcomes. While Higher 
Achievement had no measurable impacts in Year 1, by Year 2 there were positive impacts on the 
math problem-solving and reading comprehension test scores. By Year 4, the control group tested 
at the same level as the program group in reading comprehension, but the impact on the program 
group’s math problem-solving test score did not shrink and remained statistically significant. 


Third, Higher Achievement had an impact on the types of high schools its scholars ultimately 
attended. Because the schools varied widely in rigor and focus — particularly the magnets and 
charters — Higher Achievement identified the stronger, more competitive schools and tailored 
its recommendations to each scholar’s interests and strengths. Competitive magnet schools were 
defined as those with an explicit academic focus, selective admissions, and with strong offerings 
of Advanced Placement and Honors classes. Competitive charter schools were defined as those 
that DC Public Schools had classified as Tier 1 (the academically best schools) in their three-tier 
classification system, based on such measures as standardized test scores and graduation rates. 
The research team used the same classification system as did Higher Achievement to designate a 
public high school as either competitive or not competitive. Unfortunately, private schools were 
not classified by any system. At a national level, private schools tend to have smaller class sizes 
and more rigorous academic programs than public schools; seniors enrolled in them are more 
likely to go to college, even after accounting for differing aspirations, abilities, and socioeco- 
nomic status.'? Therefore, all private schools were grouped together into their own category.” 


Analysis of high school matriculation rates found that program students were more likely than 
control group students to matriculate to private schools (15 percent versus 9 percent, respectively) 
and less likely to go to noncompetitive charters or magnets (6 percent versus 13 percent). There 
was no impact on attending competitive charters or magnet schools (about 40 percent for both 
groups), and no impact on attending a neighborhood public school (about 35 percent for both 


11 These rates reflect the participation of the 81 percent of students who completed the tests and surveys 
in Year 2. It does not reflect the few students in the study who declined to participate in the follow-up 
surveys. Since those students who declined were probably more likely to drop out than those who agreed 
to participate, these rates may slightly overestimate the program’s overall participation rate. 


12 If students enrolled in Higher Achievement as sixth-graders, four years after random assignment they 
would be in ninth grade and no longer in the program. Thus, the 47 percent participation rate is based 
only on those youth who were still eligible to participate in the program four years later, namely those who 
started the program as rising fifth-graders. 


13 Choy (1997); Falsey and Heyns (1984). 

14 Higher Achievement staff, who knew the area high schools well, designated 50 of the 58 private/parochial 
schools as either elite (including 14 nationally recognized schools such as Georgetown Prep or Sidwell), 
or academically competitive (36). Because the vast bulk of the scholars who applied to a private school 


applied to one of the 36 academically competitive schools, all private schools were pooled into a single 
category. Linden, Herrera, and Grossman (2013). 
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groups). These impacts mimic the impacts on the types of schools to which scholars applied. 
Program students applied more often to private schools and less often to noncompetitive charters 
and magnets. In sum, Higher Achievement seemed to get scholars to choose private schools over 
noncompetitive charters and magnets in their high school applications. Box 1 summarizes the 
key findings from these early studies of the program. 


THE LONG-TERM FOLLOW-UP STUDY: 
IMPACTS ON THE TYPES OF COLLEGES ATTENDED 


How might Higher Achievement, a program serving youth in middle school, affect the types 
of colleges the participants attend? Higher Achievement applicants (the research sample) are a 
motivated group of students: All are quite likely to go to college, with or without the additional 
support. However, the program may very well affect the quality of the colleges these students 
attend. Figure 1 shows how Higher Achievement could help students go to academically stronger 
colleges that are better matched to their abilities. 


First, its academic instruction aims to strengthen scholars’ middle school academic performance, 
which is especially important for doing well on high school entrance exams and for being ad- 
mitted to more competitive high schools. In addition, if their high school grade trajectory stays 
strong, students can be candidates for academically stronger colleges."® 


Second, scholars participate in activities to learn about the array of local college-preparatory 
high schools and academically strong colleges by visiting and talking to people in these insti- 
tutions. These activities should result in scholars being more interested in applying, and thus, 
being admitted to high schools that are more competitive with respect to college admission. 


Third, eighth-grade scholars practice for and take standardized tests — the Secondary School 
Admissions Test (SSAT) and/or the High School Placement Test (HSPT) — that are very similar 
in structure to the SAT and ACT. This practice should not only help their performance on the 
high school entrance exams but also give them the skills they'll need in the future to study for the 
SAT or ACT. In addition, the scholars’ parents are guided through the financial aid application 
process and learn that a school’s stated tuition is not necessarily what they have to pay. At the 
same time, they get experience applying for financial aid — a skill that can be applied to the 
college financial aid process further down the road. In other words, while not a direct goal of 
Higher Achievement, the program hopes that the experiences it provides families in respect to 
the high school application process will make choosing and applying to academically competitive 
colleges less daunting. 


All of Higher Achievement’s program activities are aimed at increasing the likelihood that 
scholars will end up going to academically strong college-preparatory high schools. From there, 
scholars should not only have access to better college guidance services, but also become better 


15 Nagaoka, Roderick, and Coca (2009). 
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BOX 1 
Key Findings from Earlier Reports on Higher Achievement: 
Scholars’ Outcomes 


SCHOLARS’ MIDDLE SCHOOL STANDARDIZED TEST SCORES COMPARED WITH 
CONTROL GROUP 


Year 2 


Math problem-solving average: 0.10 SD higher 
Reading comprehension average: 0.08 SD higher 


Year 4 


Math problem-solving average: 0.11 SD higher 
Reading comprehension average: 0.04 SD higher and statistically significant 


SCHOLARS’ HIGH SCHOOL APPLICATION BEHAVIOR COMPARED WITH CONTROL GROUP 


More likely to gather data about their preferred high schools and to practice for the SSAT and 
HSPT, by 17 percentage points* 


More likely to take the SSAT and HSPT, by 7 percentage points 


More likely to apply for financial aid and scholarships to cover high school tuition, by 17 percentage 
points 


More likely to receive financial aid and scholarships to cover high school tuition, by 6 percentage 
points 


HIGH SCHOOL MATRICULATION AMONG SCHOLARS COMPARED WITH CONTROL GROUP 
More likely to attend private schools, by 6 percentage points 


Less likely to go to noncompetitive charters or magnets, by 7 percentage points 


NOTE: Because the units of test scores differ across districts, it is standard to “normalize” the 

test scores — that is, to put test scores in terms of acommon unit, namely standard deviation 

(SD) units — and to center each student’s test score measure around the mean [score] of the 
assessment. The SD of test scores is the average number of points between a student’s score and 
the mean score on the assessment. Thus, the normalized score for a student who tested at the 
average score of the assessment would be 0.0. The impact on the Year 2 reading of 0.10 in this box 
means that the program group students, on average, scored higher than the control group students 
by one-tenth of a standard deviation on their standardized assessment. 


“Scholars gathered data about high schools by, for example, making visits and talking to staff or 
students at those schools. The SSAT and HSPT are standardized high school entrance exams. 
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prepared academically to do college coursework. In fact, an analysis of the National Education 
Longitudinal Study (NELS) shows that students who graduate from private or parochial high 
schools are 46 percent and 36 percent, respectively, more likely to earn a bachelor’s degree, even 
after controlling for socioeconomic status and test scores." 


To test if Higher Achievement’s particularly intensive middle school intervention was sufficient 
to change the types of colleges its scholars attended, the research team obtained students’ college 
enrollment records from the National Student Clearinghouse (NSC) in the summer of 2019 to 
access information about college attendance through the spring of that year. By then, all stu- 
dents in the sample should have had at least three years after high school to enroll in college.” 


The NSC maintains enrollment records for almost all students enrolled in colleges in the United 
States. By matching the study sample to the NSC records, the research team obtained the college 
enrollment information for the students in the study.'® The records included when the students 
were enrolled in a college, the name of that college, and whether it was a two- or four-year in- 
stitution.’? Over time, students could attend more than one college. If an NSC record was not 
found for a student, the study categorized the individual as not having yet attended college. This 
is the standard assumption used in all NSC-based studies. A recent methodological study found 
that NSC records capture 99 percent of public and nonprofit private college enrollment, but the 
data will somewhat understate the college attendance of the sample.”° 


For the purposes of this short report, the team defines whether a college is higher quality in 
two ways. First, four-year colleges are deemed higher quality than two-year colleges because 
students are much more likely to earn a degree at four-year colleges and the degrees they earn 
are more highly valued in the labor market.*! Second, a college is defined as higher quality if 
it is academically more competitive, as measured by higher average SAT scores of its incoming 


16 Bozick and DeLuca (2005). 


17 Because the study enrolled fifth- and sixth-graders in waves over three years (2006, 2007, and 2008), 
sample members had had different amounts of time to have attended college. The students with the 
longest time were the sixth-graders who entered in 2006. They should have graduated high school in 
spring 2013 and have had six years from high school graduation to matriculate to college by 2019. The 
students with the shortest period to have enrolled in college were the fifth-graders who enrolled in the last 
year of study intake, 2008. They should have graduated high school in 2016 and have had three years to 
matriculate prior to 2019. Many students from low-income schools do not go to college immediately after 
high school but work to earn money for college; National Student Clearinghouse Research Center (2019). 
Thus, it was important to wait until 2019 to allow the youngest members of the study sample time to enroll. 


18 To match the research sample to their data, the team provided the NSC with students’ names and 
birthdays. 

19 For some schools, the NSC can provide more information, such as major, degree type, and so on. 
However, few schools report this information. The research team lacked this information for too many 
subjects in the sample to be able to use what it did have for the analysis. 

20 Dynarski, Hemelt, and Hyman (2015). This study concludes that the NSC “misses a shrinking but nontrivial 
portion of undergraduate enrollment” (p. 25), capturing 99 percent of enrollment in public and private 
institutions but only 48 percent of for-profit college enrollment. 


21 Shapiro et al. (2019); PayScale (2009) . 
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first-year students or by lower acceptance rates.*? The student body in competitive colleges is 
academically more able and thus classes are more rigorous. Employers value these colleges more 
because they teach students how to master new and difficult skills, tasks that are useful in the 
business world and in life.?° 


QUESTIONS AND FINDINGS 


The long-term follow-up portion of the study examined the following research questions: 


e Were students with access to Higher Achievement (program group) more likely to 
go to four-year colleges than control students? 


e Were program students more likely to go to more selective, academically stronger 
four-year colleges than control students? 


The students who applied to Higher Achievement during the study were academically motivated 
students from families that valued education enough to commit to sending their child to three 
or four years of Higher Achievement’s Summer and Afterschool Academies. Because of that, the 
research team expected almost all of the sample students (both program and control) to attend 
college. To be sure, college enrollment is no longer the challenge that it once was; 86 percent of 
high school graduates start at either two- or four-year colleges within eight years.2* However, 
academically motivated students from disadvantaged backgrounds frequently “under-place.” 
That is, rather than attending selective, academically competitive schools for which they are 
qualified, they go to much lower-ranked colleges because of a lack of information about more 
competitive schools and how to get into them.* Thus, this study examined the more interesting 
questions related to the quality of colleges that program students attended. 


This study’s basic conclusion is that Higher Achievement did not impact the type of college 
scholars attended. Figure 2 shows the percentage of program and control group students who 
ever attended two-year versus four-year colleges, as well as the percentage of those who attended 
any type of college, for context. First, note that the college-going rate was quite high and similar 
for both groups, as expected, at approximately 70 percent. However, the hypothesized differ- 
ence in college type did not appear. By 2019, approximately 60 percent of both program and 
control students had ever attended a four-year college and 23 percent had ever attended a two- 
year college. (Some students switched between the two over the study period and so are in both 
counts.) None of these rates differs statistically between program and control students. These 


22 To categorize colleges as more or less academically strong, the average SAT scores of incoming first-year 
students and acceptance rates were taken from the colleges’ websites and merged into the dataset. 


23 Mulhere (2019). 
24 Rosenbaum, Ahearn, Becker, and Rosenbaum (2015). 
25 Bowen, Chingos, and McPherson (2009); Hoxby and Turner (2013); Roderick, Nagaoka, and Coca (2009). 
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FIGURE 2 Rates of Ever Attending College, Overall and by Type 
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SOURCE: MDRC calculations based on the regression described in the Table 1 note. 


NOTE: None of the impacts are statistically significant at a .10 level. 


small numerical differences in the averages are likely to have occurred by chance.” While not 
shown, the percentages of program and control students who first attended a four-year college 
were also about the same: Fifty-seven percent of the program students and 56 percent of the 
control students attended a four-year college as their first college. As before, the difference in 
these rates is not statistically significant. 


26 The impact estimates come from regressions on the outcomes of 521 program students and 430 control 
students. The following baseline covariates were included in each regression: baseline scores for reading 
and math; self-perceptions of academic abilities — industry and persistence; creativity, enjoyment of 
learning, curiosity, and ability to change the future through effort; a measure of peer academic support; 
a measure of general adult support; dummy variables for grade at baseline; age; an indicator variable for 
receipt of free or reduced-price lunch; dummy variables for race; a dummy variable for the year the child 
applied to Higher Achievement; and a dummy variable for whether the student applied with a sibling. In 
addition, there are family-level controls including fixed effects for the center to which the family applied, 
parent’s education, household income, household composition, and an indicator for whether English was 
the primary language spoken at home. 
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Next, the research team examined the academic competitiveness of the schools that college-going 
program group students attended, to see if they went to more rigorous schools than those at- 
tended by college-going control students. Figure 3 shows the quality of the best college attended 
in three ways: the average math SAT score of the freshman class, the average reading/writing 
SAT score of the freshman class, and the acceptance rate. Again, there were no statistically sig- 
nificant differences between the program and control students. Among the college-goers, study 
participants (both program and control students) attended colleges where the average math 
SAT score of incoming freshmen was 542 (out of 800), while the average reading and writing 
score of incoming freshmen was 553 (out of 800). The overall acceptance rate at these colleges 
was 63 percent. 


FIGURE 3 Academic Selectivity of the Best College Attended, as Measured by 
the Average Math and Reading SAT Scores of Incoming Freshmen and by the 
Acceptance Rate 
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NOTE: None of the impacts is statistically significant at the 0.10 level. 


Digging deeper, the research team investigated whether Higher Achievement was more effective 
for scholars whose parents had not attended college or whose family income at baseline was in 
the lower half of the applicant pool, given that the decisions of these families might be most 


14 | THE IMPACT OF AN INTENSIVE YEAR-ROUND MIDDLE SCHOOL PROGRAM ON COLLEGE ATTENDANCE 


influenced by the knowledge of how to apply for financial aid that is taught in the eighth grade. 
The team also examined whether the program was more or less effective for various marginal- 
ized groups — female students, non-English-speakers, and Black and Latino youth — relative 
to students of other races and ethnicities. No statistically significant differences were found.?’ 


CONCLUSION 


The results of the long-term follow-up study found that the earlier impacts were not large enough 
to change the type of colleges scholars attended. In both the program and the control group, 
attendance at four-year colleges was quite high, approximately 60 percent. The selectivity of 
the colleges that the two groups attended was also very similar: They accepted 63 percent of 
applicants and had student bodies with very similar math and reading SATs. 


The results of the college follow-up study are disappointing, but perhaps not surprising; the 
vast majority of the students in both groups were attending similar types of high schools. Just 
a small percentage of the Higher Achievement scholars were attending schools that would be 
expected to have better college placement services than those offered in the control group’s 
high schools. Thus, the likelihood that program group students would choose a college that 
undermatched their academic ability (in terms of academic competitiveness or selectivity) was 
about the same as it was in the control group. While many selective and academically competi- 
tive colleges actively try to increase their diversity, they generally do so by recruiting at only a 
handful of elite city schools.*8 For these reasons, programs like Higher Achievement may want 
to connect students with interventions like College Match that explicitly help students better 
match to colleges, or with organizations that work directly with colleges to help broaden the 


sources from which they recruit.?9 


Although Higher Achievement did not affect the quality of the colleges its scholars attended, it 
may have provided them with social and emotional skills that could affect more subtle aspects 
of their college experience and life success: for example, the courses they take, the grades they 
receive in those courses, how engaged they are in the learning process, how comfortable they are 
in the college environment, the extent to which they get involved in additional academic and ex- 
tracurricular experiences, and the types of jobs they obtain later. Career Academies, for example, 
serve a similar set of motivated students.°° Delivered in high schools, Career Academies offer 
participants small, themed learning environments and expose them to their chosen careers for 
four years. An evaluation of that intervention found that students in the program group graduated 
at exactly the same rate as students in the control group, and they also attended college — and 
the same types of colleges (two-year, four-year, and certificate programs) — at the same rate. In 


27 Asall of the earlier studies found, Higher Achievement appears to affect or not affect students of all types 
similarly. 


28 Vedantam (2013). Not presented here, but supplemental analysis found Higher Achievement did not have 
an impact on the percentage of students who matriculated to Washington, DC’s elite high schools. 


29 Byndloss et al. (2015). 
30 Kemple and Willner (2008). 
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the longer term, however, program students were more likely to be employed in higher-paying 
jobs and thus had higher earnings. Only additional follow-up of Higher Achievement’s scholars 
will reveal if it, too, has such benefits. 


Of course, no program can or should be expected to be all things to all young people during their 
critical developmental years. While Higher Achievement did not change the college trajectory of 
its scholars, this evaluation found that the program was very engaging to middle school students 
and improved their academic performance. It also helped some of its scholars get into better high 
schools. In the years since the beginning of this evaluation, Higher Achievement has strengthened 
its programming, and a recent study finds it has even larger impacts on course grades two years 
after enrollment than it had on test scores in the original study.*' Given that recent literature 
shows that course grades are more strongly related to future earnings than test scores, this is 
good news for current participants.* Still, it is extremely rare for short-term youth programs 
by themselves — even multiyear, intensive programs like Higher Achievement — to have long- 
term impacts. Rather, programs serving young people in under-resourced neighborhoods need 
to help them and their families build a mosaic of strong in- and out-of-school experiences with 
caring adults that shift with each child’s interests and developmental needs. Selected activities or 
programs should be challenging, yet engaging, stretching the young person in developmentally 
appropriate ways. It is this type of rich, developmentally responsive environment that, over time, 
supports young people to achieve their full potential. 


31 Garcia et al. (2020). The original evaluation collected data on test scores only, not on course grades. 
32 Heckman and Kautz (2012). 
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ABOUT MDRC 


MDRC IS A NONPROFIT, NONPARTISAN SOCIAL AND EDU- 
CATION POLICY RESEARCH ORGANIZATION DEDICATED TO 
learning what works to improve the well-being of low-income 
people. Through its research and the active communication of its 
findings, MDRC seeks to enhance the effectiveness of social and 
education policies and programs. 


Founded in 1974 and located in New York; Oakland, California; 
Washington, DC; and Los Angeles, MDRC is best known for 
mounting rigorous, large-scale, real-world tests of new and ex- 
isting policies and programs. Its projects are a mix of demon- 
strations (field tests of promising new program approaches) and 
evaluations of ongoing government and community initiatives. 
MDRC’s staff members bring an unusual combination of research 
and organizational experience to their work, providing expertise 
on the latest in qualitative and quantitative methods and on pro- 
gram design, development, implementation, and management. 
MDRC seeks to learn not just whether a program is effective but 
also how and why the program’s effects occur. In addition, it tries 
to place each project’s findings in the broader context of related 
research — in order to build knowledge about what works across 
the social and education policy fields. MDRC’s findings, lessons, 
and best practices are shared with a broad audience in the policy 
and practitioner community as well as with the general public and 
the media. 


Over the years, MDRC has brought its unique approach to an 
ever-growing range of policy areas and target populations. 
Once known primarily for evaluations of state welfare-to-work 
programs, today MDRC is also studying public school reforms, 
employment programs for ex-prisoners, and programs to help 
low-income students succeed in college. MDRC’s projects are 
organized into five areas: 


e Promoting Family Well-Being and Children’s Development 
e Improving Public Education 

e Raising Academic Achievement and Persistence in College 
e Supporting Low-Wage Workers and Communities 

¢ Overcoming Barriers to Employment 


Working in almost every state, all of the nation’s largest cities, 
and Canada and the United Kingdom, MDRC conducts its proj- 
ects in partnership with national, state, and local governments, 
public school systems, community organizations, and numerous 
private philanthropies. 


